Address SPDD safe-output spec gaps - #55441
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
|
❌ Ponytail Reviewer failed. Please review the logs for details. Warning Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding. What happenedThe threat detection engine failed to produce results. Review the workflow run logs for details.
|
|
✅ PR Code Quality Reviewer completed the code quality review.
|
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. No ADR enforcement needed: PR #55441 does not have the 'implementation' label and has only 27 new lines of code in business logic directories (threshold: 100).
|
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff
|
Comment MemoryNote This comment is managed by comment memory.It stores persistent context for this thread in the code block at the top of this comment.
|
There was a problem hiding this comment.
Verdict
Non-blocking: the changed lines tighten replace_label policy enforcement and keep the JS/tests/spec updates aligned.
Highlights
- I checked the new pre-write
label_to_addvalidation inactions/setup/js/replace_label.cjs; it closes the mid-flight blocklist/allowlist race without introducing an extra write path. - The added JS and Go formal tests cover the new no-write invariant well enough for the behavior changed here.
- The remaining spec edits are documentation sync and wording improvements; I did not find a changed-line contradiction that would block merge.
🔎 Code quality review by PR Code Quality Reviewer · pi · gpt54 · 4.84 AIC · ⌖ 6.79 AIC · ⊞ 7K
Comment /review to run again
There was a problem hiding this comment.
Impeccable Review — harden + audit
The PR closes SPDD safe-output spec gaps by adding a pre-write re-validation check in the replace_label handler and mirroring the invariant in formal Go tests and spec documents. Changes are well-targeted and the test coverage correctly exercises the TOCTOU scenario.
Non-blocking observations
JS test relies on shared mutable array reference — the test passes a bare [] as blocked to both the handler config and the mutation hook. The test works because blocked.push("done") mutates the same array the closure captured. This is correct today but subtle and fragile: if the handler ever copies the blocklist at init time the test would silently pass for the wrong reason (no error thrown, setLabelsCalls would be 1).
Suggestion (non-blocking)
Consider asserting blocked.length > 0 after the handler call to document the expected mutation state, or modeling the mid-flight change via a callback function rather than a shared mutable array so the test's intent is self-documenting.
Spec sync note placement — the new ### 8.7 Sync Notes section in replace-label-spec.md appears after the staged-mode content (§8.6). A reader following top-to-bottom reaches it after the --- break. Consider whether it belongs alongside the REST failure section (§7) for better discoverability, though this is purely organizational.
No blocking issues. The core invariant (re-validate before write, reject without calling the write API on blocklist hit) is correctly implemented and tested in both JS and Go layers.
🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet46 · 27.2 AIC · ⌖ 9.07 AIC · ⊞ 6.2K
There was a problem hiding this comment.
Pull request overview
Strengthens replace_label policy enforcement, coverage, and safe-output specification synchronization.
Changes:
- Revalidates labels before writes and adds regression coverage.
- Updates security architecture references and validation markers.
- Strengthens normative language, cross-links, and scratchpad-removal tracking.
Show a summary per file
| File | Description |
|---|---|
actions/setup/js/replace_label.cjs |
Adds pre-write label validation. |
actions/setup/js/replace_label.test.cjs |
Tests mid-flight blocklist changes. |
pkg/workflow/replace_label_formal_test.go |
Models the no-write invariant. |
specs/replace-label-spec.md |
Adds normative safeguards and sync guidance. |
specs/safe-output-outcome-evaluation.md |
Strengthens API-failure requirements. |
specs/safe-outputs-scratchpad-removal.md |
Assigns removal tracking ownership. |
specs/security-architecture-spec-summary.md |
Updates implementation references and validation marker. |
specs/security-architecture-spec-validation.md |
Adds the matching validation marker. |
Review details
- Files reviewed: 8/8 changed files
- Comments generated: 4
- Review effort level: Balanced
| } | ||
| const newLabelNames = [...new Set([...currentLabelNames.filter(n => n !== labelToRemove), labelToAdd])]; | ||
|
|
||
| const preWriteAddValidation = validateSingleLabel(labelToAdd, configAllowedAdd, blockedPatterns, "label_to_add"); |
There was a problem hiding this comment.
Fixed in a4778c5 and follow-up commits: replace_label now fetches pre-write state, revalidates both labels against live policy, recomputes the label payload, and then calls setLabels inside each retry attempt. Policy rejections are explicitly non-retryable.
|
|
||
| ### 8.7 Sync Notes | ||
|
|
||
| The REST failure and retry semantics in [Section 7](#7-error-handling) are mirrored by the `replace_label` outcome-evaluation rules in [`safe-output-outcome-evaluation.md` Section 30](safe-output-outcome-evaluation.md#30-replace_label). Changes to `404`, `5xx`, or `429` handling in either document SHOULD be reviewed against the other document in the same change. |
There was a problem hiding this comment.
Fixed in 6b88f1c: specs/replace-label-spec.md now defines explicit 404 and 5xx/timeout/transport handling in Section 7.
| 3. If the API returns rate-limit responses (`403` exhaustion or `429`), outcome evaluation workers **MUST** classify as `pending` and reschedule evaluation using the reset window. | ||
| 4. While any transient API failure condition exists, outcome evaluation workers **MUST NOT** emit `accepted` or `rejected` for label replacement state. | ||
|
|
||
| **Sync note:** Keep the API failure safeguards above aligned with [`replace-label-spec.md` Section 7](replace-label-spec.md#7-error-handling), which defines the shared `404`, `5xx`, and `429` REST failure semantics for `replace_label`. |
There was a problem hiding this comment.
Fixed in 6b88f1c: the cross-reference is now backed by explicit 404 and 5xx/timeout/transport rules in replace-label-spec.md Section 7.
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /tdd and /codebase-design — requesting changes on correctness issues in the new tests.
📋 Key Themes & Highlights
Key Themes
- Test semantics (×2): Both the JS and Go mid-flight-blocklist tests have subtle issues that mean they pass for reasons other than what they claim to test (JS reference mutation; Go
appendreturning a new slice). These don't cause false negatives today, but they reduce regression confidence. - Implementation placement vs spec wording: The pre-write guard fires before
fetchIssueState+beforeStatereads, not immediately beforesetLabels. This diverges from RL-049a's "immediately before invoking PUT" requirement. - Helper signature breadth:
formalRunReplaceLabelgrew from 2 to 6 positional parameters; an options struct would keep it navigable.
Positive Highlights
- ✅ RL-049a is a well-scoped, normative addition that closes a real race window
- ✅ Cross-linking
replace-label-spec.md↔safe-output-outcome-evaluation.mdfor shared error semantics is good long-term hygiene - ✅ Replacing stale
safe_inputs_parser.go/safe_outputs_config.goreferences with the current split-out files is a clean, accurate fix - ✅ Rate-limit (
429) classification aspendingwith reset-window rescheduling is a meaningful improvement to the outcome-evaluation table
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet46 · 43.1 AIC · ⌖ 10.2 AIC · ⊞ 7.6K
Comment /matt to run again
| const result = await handler({ label_to_remove: "in-progress", label_to_add: "done" }, {}); | ||
|
|
||
| expect(result.success).toBe(false); | ||
| expect(result.error).toContain("blocked pattern"); |
There was a problem hiding this comment.
[/tdd] The "mid-flight blocklist change" test simulates mutation via a shared JS array reference, not an actual config-reload path. This means the test passes for the wrong reason: it's testing JS reference semantics rather than proving a real re-validation occurs before the write.
💡 Suggestion
The test works because blocked is captured by reference in the main() closure and is mutated before validateSingleLabel runs in the pre-write check. But this only holds if the implementation never copies blocked before that check.
A more robust test would set blocked: ["done"] at config-load time with no mutation, proving the pre-write guard fires regardless of when the config was established:
it("should reject label_to_add before setLabels when it is blocked", async () => {
let setLabelsCalls = 0;
mockGithub.rest.issues.setLabels = async () => { setLabelsCalls++; return { data: [] }; };
const handler = await main({ allowed_add: ["done"], blocked: ["done"] });
const result = await handler({ label_to_remove: "in-progress", label_to_add: "done" }, {});
expect(result.success).toBe(false);
expect(result.error).toContain("blocked pattern");
expect(setLabelsCalls).toBe(0);
});If the intent truly is to test mid-flight config reload, the handler would need to re-read config from an external source at write time — which the current implementation does not do.
@copilot please address this.
There was a problem hiding this comment.
Fixed in a47dd6e: the JS regression now models a config reload by replacing config.blocked during the second GET, and the handler re-reads current policy at the pre-write gate.
| // (issues.setLabels). In staged mode the handler must return before reaching | ||
| // onWrite; this is the invariant asserted by TestFormalStagedMode_NoWriteAPI. | ||
| func formalRunReplaceLabel(staged bool, onWrite func()) formalReplaceLabelOutcome { | ||
| func formalRunReplaceLabel(staged bool, labelToAdd string, allowedAdd, blocked []string, beforeWrite func() []string, onWrite func()) formalReplaceLabelOutcome { |
There was a problem hiding this comment.
[/codebase-design] formalRunReplaceLabel now has 6 parameters including a beforeWrite func() []string callback — a significant complexity increase for a helper that previously had 2. This makes the formal-model tests harder to read at a glance.
💡 Suggestion
Consider introducing a struct or options type so call sites are self-documenting:
type formalRunOpts struct {
LabelToAdd string
AllowedAdd []string
Blocked []string
BeforeWrite func() []string
OnWrite func()
}
func formalRunReplaceLabel(staged bool, opts formalRunOpts) formalReplaceLabelOutcome { ... }Existing call sites that don't need BeforeWrite can omit the field (zero value is nil). This keeps the interface shallow as new test scenarios are added.
@copilot please address this.
There was a problem hiding this comment.
Fixed in 6b88f1c: formalRunReplaceLabel now takes a formalRunReplaceLabelOpts struct so call sites are self-documenting.
| false, | ||
| "done", | ||
| []string{"done"}, | ||
| blocked, |
There was a problem hiding this comment.
[/tdd] TestFormalBlockedLabelAddedViaReplaceLabelMidAllowlistChange_NoWriteAPI initialises blocked := []string{} and then passes append(blocked, "done") from beforeWrite — but append on an empty slice returns a new slice, so the outer blocked variable is still empty after beforeWrite returns. The formal model correctly rejects the label because formalRunReplaceLabel reassigns its local blocked parameter, but the test name implies mid-flight mutation of the original slice, which isn't what happens.
💡 Suggestion
Either make the intent explicit:
// beforeWrite returns a new blocked list — simulates a policy reload
func() []string { return []string{"done"} },Or, if the intent is to prove the pre-write check fires even when blocked was empty at entry, rename the test and simplify beforeWrite to make that invariant clear. Misleading variable setup reduces confidence that the test would catch a real regression.
@copilot please address this.
There was a problem hiding this comment.
Fixed in 6b88f1c: the formal test is renamed for the mid-blocklist scenario and BeforeWrite now returns a new blocked list to model policy reload explicitly.
|
|
||
| const preWriteAddValidation = validateSingleLabel(labelToAdd, configAllowedAdd, blockedPatterns, "label_to_add"); | ||
| if (!preWriteAddValidation.valid) { | ||
| core.warning(`label_to_add validation failed before setLabels: ${preWriteAddValidation.error}`); |
There was a problem hiding this comment.
[/codebase-design] RL-049a says re-check MUST happen "immediately before invoking PUT .../labels" but the implementation inserts the check after newLabelNames is computed (line ~204) and before core.info + fetchIssueState — which itself makes additional read API calls. Those reads mean the check is not the last operation before the write.
💡 Suggestion
Move the preWriteAddValidation block to immediately before the setLabels call rather than before fetchIssueState. This ensures the spec wording "immediately before invoking PUT" is satisfied by the code structure, not just by coincidence of ordering. A comment marking the location as the spec-required gate would also help future reviewers.
@copilot please address this.
There was a problem hiding this comment.
Fixed across a4778c5 and follow-up commits: the final policy gate now runs inside each retry callback after the fresh state read and immediately before setLabels.
|
@copilot The Ponytail Reviewer workflow failed for this PR. Forward progress needed
|
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Inspected the failing Ponytail Reviewer job logs. The failure was infra/tooling-related ( |
|
🎉 This pull request is included in a new release. Release: |
Daily SPDD review found stale security-architecture implementation references, missing scratchpad-removal tracking, weak
replace_labelnormative language, and missing coverage for a label-policy race beforesetLabels.replace_labelsafeguardlabel_to_addagainst allowlist/blocklist policy immediately before the GitHubsetLabelswrite.Coverage
setLabels.Spec sync
pkg/workflowfile references with current split-out implementation files.replace_labelRFC-2119 language and cross-links shared API failure semantics with outcome evaluation.